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1.  Introduction 


State  awareness  is  an  essential  prerequisite  for  autonomous  navigation.  Like  their 
biological  counterparts,  modem  guided  munitions  rely  on  inertial  and 
environmental  measurements  to  infer  pose,  position,  and  motion  with  respect  to 
their  surroundings.  The  Earth’s  magnetic  field,  position  of  the  sun,  global 
positioning  system  (GPS),  and  even  the  ground-sky  temperature  gradient  are  some 
of  the  environmental  cues  that  have  been  leveraged  toward  projectile  Guidance 
Navigation  and  Control  (GNC).  Only  recently,  however,  have  researchers  begun  to 
explore  and  develop  vision-based  techniques  to  aid  projectile  GNC.  This  report 
focuses  specifically  on  the  use  of  optical  flow  (OF)  in  recovering  vehicle  state. 

Just  as  points  and  objects  in  a  3-dimensional  (3-D)  scene  are  projected  onto  a  plane 
to  form  an  image,  so  are  velocity  vectors  of  these  same  points  and  objects.  This  is 
the  projected  OF  field.  While  the  flow  field  can  be  described  in  closed  form,  the 
inverse  problem  of  recovering  self-motion  from  OF  is  an  area  of  ongoing  research. 
Several  literature  reviews  describe  the  theory  and  techniques  used  to  infer  ego- 
motion  from  OF.1-3  The  mathematics  relating  self-motion  to  perceived  OF, 
developed  during  the  past  30  years,  has  been  applied  notionally  to  various 
experimental  platforms,  including  robots,4  unmanned  aerial  vehicles  (UAVs),5 
aircraft,6  and  missiles.7 

In  general,  the  possibility  of  motion  within  a  nonrigid  scene  adds  an  unknown 
number  of  degrees  of  freedom.  Fortunately,  the  projectile  application  typically 
admits  several  reasonable  assumptions  and  constraints  that  make  the  problem  more 
tractable.  In  particular,  if  the  visible  scene  is  rigid,  then  the  OF  can  be  described 
entirely  in  terms  of  the  motion  of  the  observer  and  camera  intrinsic  parameters  (e.g., 
focal  length,  distortion).  Under  this  assumption,  we  can  make  use  of  the  so-called 
continuous  differential  epipolar  constraint8  to  solve  for  the  body-fixed  translational 
velocity  (up  to  scale9)  and  angular  rates. 

The  goal  of  this  work  is  to  specialize  and  apply  the  theory  of  OF  geometry  to 
projectile  vision  based  navigation.  In  the  following,  we  describe  the  construction 
of  an  OF  simulation  tool  for  projectile  flight.  We  then  present  the  equations  and 
constraints  relating  OF  to  self-motion.  Finally,  an  example  simulation  is  presented 
wherein  we  generate  vehicle  state  and  OF  “ground  truth”  histories,  corresponding 
video  sequence,  measured  OF,  and  state  estimation  history.  The  report  concludes 
with  a  discussion  of  results  and  future  research  paths. 
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2.  Conventions 


1)  Vectors  are  bold  lowercase  (e.g.,  v).  Matrices  are  bold  uppercase  italic 
(e.g.,  R ).  Scalars  are  lowercase  italic  (e.g.,  /). 


2)  Vectors  with  homogeneous  coordinates  appear  under  the  tilde  (e.g.,  v). 


3)  [a]x  designates  the  skew-symmetric  square  matrix  associated  with  the 
vector  a  so  that  [a]xb  =  a  x  b 


[a]x  = 


0  —ak 


CL; 


CLk  0  CL  i 
—  CL;  CL;  0 


(1) 


*J  1 

4)  There  are  several  coordinates  systems  used,  defined  by  location  of  origin 
followed  by  orthonormal  basis  vectors. 


a.  Earth  frame,  VE  =  {e0,  %  y,  z).  This  is  the  gun-fixed  inertial 
frame  with  origin  e0  coincident  with  the  firing  platform.  We  define 
positive  z  pointing  toward  the  center  of  the  Earth,  x  points  downrange, 
and  y  completes  the  right-handed  basis. 

b.  Body  frame,  rs  =  {b0, 1,  j,  k}.  This  projectile-fixed  inertial 
frame  is  instantaneously  at  rest  with  respect  to  T£.  The  origin  b0  is 
coincident  with  the  optical  center  of  the  projectile-mounted  camera,  a 
distance  /  behind  the  focal  plane  (Fig.  1).  i  points  along  the  projectile’s 
long  axis  and  defines  the  optical  axis,  j  and  k  span  the  focal  plane. 


Fig.  1  Body  frame  definition  in  relation  to  image  plane 


2 


c.  Rotating  Body  frame,  T B,.  This  noninertial  frame  is  coincident 
with  rB,  but  is  subject  to  both  translational  and  rotational  differential 
motion  (with  respect  to  T E).  Practically,  this  is  the  frame  from  which 
OF  is  measured. 

5)  When  viewed  from  a  particular  frame,  a  vector  or  vector  component  may 
be  followed  by  a  superscript  denoting  the  reference  frame  (e.g.,  vB  or  vf). 
If  clear  from  the  context,  the  superscript  may  be  omitted. 

6)  A  matrix  transformation  between  frames  will  be  designated  by  subscript  and 
superscript  (e.g.,  RE  designates  a  rotation  from  VE  to  V B),  unless  clear  from 
the  context. 

3.  Projectile  Vision  Model 


State  estimation  performance  will  depend  on  imaging  constraints,  flight  behavior, 
scene  structure,  and  OF  measurement  methodology.  Table  1  lists  some  of  these 
parameters. 


Table  1  Projectile  vision  model  parameters 


Projectile 

Input 

Camera 

Environment 

Output 

• 

Quadrant  Elevation  (QE) 

•  frame  rate 

•  Scene  structure 

• 

12  state  history 

• 

Initial  velocity 

•  focal  length 

(point  cloud) 

• 

Imaging  history 

• 

roll  rate 

•  field  of  view 

• 

Optical-flow 

• 

max  pitch/yaw 

•  imaging 

history 

perturbations 

resolution 

• 

peak  angular  rates 

In  order  to  develop  an  OF  simulation  tool,  we  must  formulate  the  component 
models  for  projectile  flight,  imaging,  and  OF  in  terms  of  these  parameters.  The 
remainder  of  this  section  details  the  construction  and  integration  of  these  models. 

3.1  Flight  Model 

For  vision-based  applications,  we  are  concerned  with  the  image-space  response  to 
specified  flight  dynamics  rather  than  with  aerodynamic  forces  and  moments.  We 
therefore  adopt  a  simple  flight  model,  which  extends  the  point  mass  model  by 
imposing  prescribed  pose-perturbations  about  the  velocity  vector. 


3 


The  translational  motion  is  given  by  the  point  mass  vacuum  model: 

170COS 

vE[t]  =  y0cos[QE]  sin[i/>] 
gt  —  v0  sin[QE] 


(2) 


r  E[t]  = 


t  cos[QE]  cos [i/j]v0  +  Xq  n 
t  cos[QE]  sin[xp]v0  +  y% 


gV_ 

L  2 


-  t  sin[QE]v0  +  Zq 


(3) 


The  projectile’s  long  axis  (l)  is  nominally  oriented  along  v.  The  pitch  and  yaw 
angles  associated  with  v  are,  respectively: 


0V  =  sin  1 


x/>v  =  tan 


v  •  z 


v-yi 


v  •  X 


(4) 


(5) 


The  pitch  perturbation  0p  is  taken  to  be  a  sinusoidal  function  subject  to: 

%  =  ft)_  C0SM  •  («) 

e,,[oj  =  o .  (7) 

Where  (0p)  and  Y  are  constants.  The  solution  to  Eq.  6  is: 

TTLUX 

iPv) 


0V  = 


'max 


sin[yt] . 


(8) 


From  Eq.  8,  it  can  be  seen  that  the  prefactor 


angle  (dp)  from  v  so  that 

TYICLX 

6„  =  (OS)  sin 

v  v  v  J  max 


ow. 


Op  =  ( dp )  cos 

y  v  y  J  max 

Similarly,  for  yaw  perturbation,  we  have 

^ P  =  (&p)mnr  ** 


(.Op) 

(.Op) 


:  is  the  maximum  pitch  deflection 


(9) 


max 


(Op) 


max  J 


(10) 


0 K) 


max 


(V'p) 


max  J 


(ID 


4 


4  =  s[StH-  <12) 

) max  J 

The  roll  angle  0  will  depend  on  a  prescribed  roll  rate  0  according  to 

0  =  0t .  (13) 

From  Eqs.  4,  5  and  9-13  we  construct  the  Euler  angles: 


Assuming  0V  and  0  are  constant  in  time  and  v  =  ||  v|| ,  the  angular  rates  are: 


Using  Eqs.  14  and  15  we  can  generate  example  12-state  histories  with  prescribed 
angular  characteristics.  {0, 0}  correspond  to  the  effective  slew  rates  of  the  strap- 
down  imager.  In  Eq.  15  the  first  term  in  6  accounts  for  the  rate  of  ballistic  overturn. 
Thus,  while  0  is  directly  specified  by  ipp,  0  will  differ  slightly  from  the  specified 
9p. 

3.2  State  Transformations 

Euler  angles  define  the  relation  between  the  Earth  and  body  frames.  It  will  be  useful 
to  transform  the  generated  states  from  Earth  to  body  frame,  and  vice  versa.  The 
rotation  sequence  used  here  is  the  so-called  ZYX  Aerospace  sequence1011 

nB  _  nX  r>y  nz 
KE  —  Kcj)Keti  i/j 

r  cecxp  cesxp  -s6n 

i  (  1  O  ) 

=  C^SqScP  —  CfpS^p  C^Cip  +  SgS^S^jj  CqSq  _ 

CqC^Sq  +  —C^Sfj,  +  CqSqS^  CgC(p_ 
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The  rows  of  RB  define  the  orthonormal  basis  of  the  body  frame  with  respect  to 
rE,  {!,  j,  k}  .  The  inverse  rotation  is  f?f  =  (/?f  )T,  and  the  body-fixed  translational 
velocity  vB  =  {vi,  Vj,  vk}  is  then 

vB  =  i?fvB  .  (17) 

The  body-fixed  angular  rates  ooB  =  {coj,  a )j,  cok}  are  given  by:10 


O 

1 

On 

0' 

toB  = 

0  C#  CqSq 

e 

0  —  s<p  CqCq 

A 

(18) 


The  pitch  angle  of  attack,  a,  and  the  yaw  angle  of  attack,  / 3 ,  are  defined  as 


a  =  tan  1 


vjl 
-vj  ' 


P 


(19) 

(20) 


3.3  OF  Field _ 

We  next  seek  to  define  the  flow  field  as  a  function  of  vehicle  state.  The  imager  is 
taken  to  be  a  strap-down  forward-facing  pinhole  camera.  To  be  specific,  we 
imagine  a  scene  point  p  e  E3  and  corresponding  projection  q  6  R3  on  the  focal 
plane.12  Our  goal  is  to  find  the  transformation  from  pB  to  qB  and  construct  the  OF 
field  {qB,  qB}. 


3.3.1  Projected  Scene 

To  project  a  scene  point  defined  in  r£  onto  the  focal  plane,  several  sequential 
transformations  are  needed:  1)  translation  to  body  origin,  2)  rotation  to  body  frame, 
and  3)  projection  to  focal  plane.  For  efficient  matrix  representation,  we  proceed 
using  homogenous  coordinates  and  corresponding  matrix  transformations.  The 
homogenous  coordinates  of  a  point  { x ,  y,  z }  in  R3  maps  as 

{x,  y,  z}  {x,  y,  z,  1}  .  (21) 


Translation.  The  frame  is  first  translated  by  the  displacement  vector  r[t]  from 
eo  to  bo : 


10  0  —  rx 

0  10  —ry 

0  0  1  — rz 

.0  0  0  1  - 


(22) 
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Rotation.  We  next  rotate  the  reference  basis  into  the  body  basis  using 


LO  0  0 


°1 

0 

0 

lJ 


(23) 


Projection.  For  a  point  viewed  in  the  body  frame,  the  pinhole  camera  model  defines 
the  projection  mapping  from  pB  to  qB ,  as  illustrated  in  Fig.  2. 


Fig.  2  Pinhole  camera  projection 


Retaining  the  depth  coordinate  in  q,  we  then  have: 


9?  =  A 


Qj  ~ 


=  fPj~ 
1  vf’ 


.pf 


q*=ft 


(24) 


In  matrix  form,  the  projective  transformation  is  then 


'  1  ooo- 
^0100 
pf  0  0  10 

1  [l/f  0  0  0. 


(25) 


Where  pf  is  the  i  component  of  pB  =  RTpE .  The  complete  transformation  for  a 
point  pE  is: 

qB  =  PRTpE  .  (26) 

Where  q  is  of  the  form  q  =  {/,  qj,  qk,  l),  and  only  [qj,  qk)  are  needed  to  form  the 
image  in  the  focal  plane. 


3.3.2  Projected  Velocity  Field 

As  we  shall  now  introduce  motion  effects  into  the  projection  model,  we  will  need 
to  distinguish  between  rotating  and  nonrotating  body  frames.  The  primed  body 
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frame  rB/  shall  be  the  rotating  (and  translating)  frame,  while  the  unprimed  rB  is 
instantaneously  at  rest  with  respect  to  rB  and  coincident  with  YB'.  The  projected 
velocity  is  measured  in  the  rotating  frame  and  is  found  by  differentiating  the  pinhole 
camera  Eq.  24,  leading  to:13 


fjPjPi  !  fjP j\ 
Pi  Pi  ) 

fkPkPi  fkPk\B 
Pi  Pi  ) 


(27) 


While  the  position  vector  is  simply  pB  =  pB,  the  derivative  pB  will  depend  on  the 
motion  of  the  viewing  platform.  The  derivative  of  a  vector  measured  in  an  inertial 
frame  is  related  to  the  corresponding  derivative  in  the  co-located  rotating  frame 
by:14,15 


p8  =  pe'fft)Xpfvs.  (28) 

Where  v  =  b0  is  the  translational  velocity  of  the  projectile  with  respect  to  rB, 
and  co  is  the  angular  velocity  of  rB'  with  respect  to  TB.  Noting  that  pB  =  0  and 
rearranging  Eq.  28  gives  the  time  derivative  of  a  scene  point  p  as  viewed  from  the 
projectile: 


p^  =  -vB  -  CO  X  p , 


PiB'  =  {-Vi  -  PkMj  +  PjO)k)B 
PjB'  =  {~Vj  +  Pk^i  -  Pi(Ok )  ■ 

PkB'  =  {-Vk  -  PjUi  +  Pi(Oj)B 
Inserting  Eq.  30  into  Eq.  27,  we  then  have:4 

fjPjVi  fjVj  fjpk0)i  fjPjPk(Oj 


Vj  = 


Pk  = 


+  ■ 


+  ■ 


Pi  Pi  Pi 
fkPkVi  fkvk  fkPj(*>i 


Pi 


Pi 


Pi 


Pi 

(  fkPV 

+  (fk  +  Jzr 

\  Pi  / 


+  ~fj  ~ 


fjPf 


(Oj- 


<Pk 

Pi  ) 

fkPjPk^k 
Pi 


(29) 

(30) 


(31) 
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Which  can  be  expressed  in  matrix  form  as 


Aj 

Ak- 


fjPj 

Pf 

fkPk 


Pi 


f± 

Pi 


-’-1  o 


0  -  — 


fk 

Pi 


fjpk  [JPJPk 


Pi 

fkpj 

Pi 


Pi 


-fj- 


fjpj 


fk  + 


fkPk 

Pi 


Pi 
fkPjPk 

pf 


vi  H 

' 

Vj 

Vk 

COi 

. 

COj 

l(oki 

(32) 


Together,  Eqs.  26  and  32  give  the  OF  field  {q,  q]  as  a  function  scene  structure  and 
projectile  motion.  For  simulation  purposes,  vB  and  ooB  are  calculated  respectively 
by  Eqs.  17  and  18,  while  pB  =  RTpE. 


4.  State  Estimation 


4.1  Direct  Differentiation  of  Perspective  Projection 


Eq.  32  shows  the  relation  between  projected  velocity  field  and  the  6  body  rates 
composing  the  linear  and  angular  velocity  vectors.  The  form  of  this  equation,  which 
directly  references  the  point  location  p,  is  useful  for  simulation  of  the  flow  vectors 
for  a  given  (known)  scene  structure  and  vehicle  state  history.  For  purposes  of  state 
estimation  where  scene  structure  is  not  typically  known  a-priori,  we  can  exchange 
{; pj,  pk}  in  favor  of  the  (measurable)  image  coordinates  [qj,  qk}.  Rearranging  the 

pinhole  camera  Eqs.  24  we  have 


PiRj  PiRk 

PI  =  JT'  Pt  =  lR 


Substituting  Eq.  33  into  Eq.  31,  we  have 
RjV[ 


Rj  ~ 


Rk 


Pi 

RkVj 

Pi 


fjVj  fj  R  k  (l>  i  RjRk^j 


Pi 

fkVk 

Pi 


fk 
fkRj^i 

~fT 


The  matrix  analog  of  Eq.  32  is  then 


Rj_ 

f± 

0 

fjQk 

Q  jRk 

^1^ 

i 

i 

r*M 

Vj 

Af 

Pi 

Pi 

fk 

fk 

Vk 

Ak- 

Rk 

-Pi 

0 

J± 

Pi 

fkRj 

fj 

Rk 

9  jtfk 

fj  J 

(Vi 

°>j 

Lo>fcJ 

(33) 


(34) 


(35) 
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Each  nth  OF  observation  {q,  q}n  then  corresponds  to  2  equations  in  7  unknowns 
(6  body  states  plus  the  scene-point  depth  pt  n).  For  N  points,  we  have  2 N  equations 
and  6  +  N  unknowns.  A  minimal  solution  would  then  require  6  points  (12  equations 
in  12  unknowns).  It  turns  out,  however,  that  vB  can  only  be  solved  for  up  to  scale; 
we  may  solve  for  the  direction,  but  not  the  magnitude  of  vB.  This  reflects  the  fact 
that  scene  depth  is  not  directly  observable  from  a  single  (monocular)  observation; 
there  is  no  way  to  decide  whether  the  scene  point  is  nearby  and  moving  slowly  or 
far  away  and  moving  quickly.  In  light  of  this  state  of  affairs,  we  may  as  well  set 
Pi  x  =  1,  which  fixes  the  scale  for  the  remaining  (unknown)  pi  n.  For  N  points,  we  now 
have  2 N  equations  and  6  +  (N  —  1)  unknowns.  The  exactly  determined  system 
requires  5  point  observations. 

Equation  35  is  nonlinear,  due  to  the  appearance  of  point  depth  pt  in  the 
denominator.  The  solution  is  not  unique,  and  several  real-valued  solutions  may 
exist.  Further  constraints  are  needed  to  identify  the  correct  solution.  Several 
reasonable  constraints  include: 

1)  Pi  >  0.  This  assumes  the  field  of  view  (FOV)  is  less  than  180°  and  asserts 
that  p  cannot  be  seen  behind  the  camera.  This  was  implicit  when  fixing 
Pi  !  =  1  but  is  true  of  all  point  depths  pi  n. 

2)  Vi  >  0.  This  asserts  that  the  projectile  never  stops  (during  flight)  or  travels 
tail-first  (with  respect  to  VE). 

3)  v  f  >  vj  A  v i  >  v\.  This  constrains  the  total  angle  of  attack  to  be  less  than 
45°. 16 

Even  with  these  constraints,  however,  our  numerical  trials  often  returned  multiple 
real  solutions,  failing  to  provide  a  robust  approach  to  state  estimation.  In  the  next 
section,  we  seek  a  stricter  physical  constraint  to  further  narrow  the  solution  space. 

4.2  Differential  Epipolar  Constraint 

Not  all  solutions  to  Eq.  35  are  physically  realizable.  Geometric  considerations  lead 
to  the  so-called  differential  epipolar  constraint,1,8  relating  the  flow  field  {q,  q)  to 
the  observer’s  motion  {v,  <*>}: 

(qT[v]xq  +  qT[<*>]x[v]xq)B  =  o .  (36) 

Where  q  and  q  are  3-vectors  of  the  form  q  =  [/,  qj,  qk],  and  q  =  [0,  qj,  qk]. 
Because  Eq.  36  is  independent  of  p,  the  number  of  unknowns  remains  exactly  6, 
irrespective  of  the  number  of  flow  vectors  considered.  To  solve  for  {v,  w},  we 
therefore  require  a  minimum  of  6  point-observations.  The  velocity  scale-ambiguity 
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is  again  manifest,  so  we  seek  only  to  recover  the  unit  vector  v  =  v/v.  Additional 
projectile  constraints  are  again  given  by: 


Vi2  +  Vj2  +  vk2  =  1 
V2  >  Vj 2 
V2  >  vk 2 
Vi>  0. 


(37) 


5.  Simulation 


State  estimation  will  rely  on  accurate  OF  measurement,  as  well  as  robust  method 
of  solution.  Both  steps  represent  potential  error  sources.  It  will  therefore  be  useful 
to  compare  performance  using  true  versus  measured  OF.  The  model  developed  in 
Section  3.3  will  generate  the  true  OF  field  for  a  given  12-state  history.  The 
measured  flow  is  derived  from  a  video  sequence  corresponding  to  the  identical  state 
history.17  An  example  of  the  corresponding  point  cloud  and  solid  geometry 
projections  is  shown  in  Fig.  3. 


Fig.  3  Projections  of  point  cloud  (left)  and  solid  geometry  onto  the  focal  plane  (right) 


The  measurement  of  OF  is  a  classical  problem  in  computer  vision  and  is  a  topic  of 
continuing  research.  We  proceed  with  a  simple  measurement  scheme  intended  to 
reflect  baseline  rather  than  optimal  performance.  Let  At  be  the  time  lapse  between 
images.  The  SURF  algorithm18  is  used  to  identify  feature  points  (q}M  in  successive 
image  frames  at  time  t  and  t  +  At.  Then  we  may  approximate  the  flow  field  as: 


{q(t  +  At/2)}M 
{q(t  +  At/2)}M 


{q(t  +  A t)}M  -  (q(t)} 


M 


At 


{q(t  +  A  t)}M  +  (q(t)} 


M 


(38) 
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Equation  38  gives  the  measured  flow  field,  analogous  to  the  true  flow  field 
expressed  in  Eqs.  26  and  32.  While  every  visible  scene  point  has  an  associated  true 
flow  vector,  only  those  feature  points  that  can  be  tracked  from  frame  to  frame  will 
have  a  measureable  flow  vector. 


Figure  4  describes  the  component  models  used  in  this  simulation. 


Projectile 


Scene 

Definition 

Scene 

Projection 


Optical  Flow 


State 

Estimation 


12  State 


Point  Cloud 


2D 

Coordinates 


Truth 


Body  Rates 


Solid 

Geometry 


2D  Image 
Sequence 


Measured 


Body  Rates 


Fig.  4  Simulation  of  projectile  state  estimation  with  exact  or  measured  OF 

Table  2  lists  the  physical  parameter  definitions  used  in  this  example  simulation. 


Table  2  Physical  parameters  used  in  simulation 


Quantity 

Value 

Units 

Description 

QE 

15 

deg 

Elevation  above  horizon 

v0 

100 

m/s 

Initial  projectile  speed 

0 op ) 

v  y  y  max 

10 

deg 

Maximum  pitch  perturbation 

^v)max 

15 

deg 

Maximum  yaw  perturbation 

4> 

2 

Hz 

Roll  rate 

(ep) 

v  y  J  max 

20 

deg/s 

Maximum  pitch  perturbation  rate 

(W max 

20 

deg/s 

Maximum  yaw  perturbation  rate 

FOV 

35 

deg 

Field  of  view 

At 

0.02 

s 

Time  lapse  between  images 

res 

3002 

pixels 

Image  resolution 

12 


6.  Results  and  Discussion 


Arcing  trajectories  can  limit  the  time  during  which  useful  imagery  can  be  captured. 
While  scene  points  are  not  constrained  to  reside  in  the  ground  plane,  this  is  typically 
the  neighborhood  where  rigid  scenery  will  be  more  plentiful  and  rich.  Thus,  higher 
launch  angle  (QE)  and  smaller  FOV  will  generally  result  in  greater  periods  of  scene 
occlusion.  Figure  5  shows  a  plot  of  points-in-view  as  a  function  of  time  for  this 
representative  simulation. 


visible  scene  points 


Fig.  5  Visible  scene  points  during  flight 

At  each  time-step,  the  epipolar  constraint  is  used  to  construct  the  least-squares 
minimization  problem 

N 

argmin^[(qT[v]xqn  +  q£[to]x[v]xqn)B]2  ■  (39) 

n=l 

Where  N  denotes  the  number  of  flow  vectors  considered,  varying  between  6  and 
15  depending  on  availability.  Equation  39  is  further  subject  to  the  motional 
constraints  of  Eq.  37. 19 

The  velocity  vector  vB,  which  can  only  be  recovered  up  to  scale,  is  used  to  calculate 
the  corresponding  angles  of  attack  via  Eqs.  19  and  20.  Results  for  pitch  and  yaw 
angles  of  attack,  as  well  as  to,  are  shown  in  Figs.  6  and  7. 
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State  Estimate  Performance 


State  Estimate  Performance 


Fig.  6  Estimate  of  pitch  and  yaw  angles  of  attack  using  perfect  OF 


Fig.  7  Estimate  of  body-frame  angular  rates  using  perfect  OF 


The  corresponding  residuals  are  plotted  by  quantile  in  Fig.  8. 


AoA  Residuals 


Fig.  8  Residuals  of  angles  of  attack  and  angular  rates  using  perfect  OF 

It  is  interesting  to  compare  these  residuals  to  those  for  the  identical  case  but  without 
the  stable-flight  constraints  of  Eq.  37,  as  shown  in  Fig.  9. 
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AoA  Residuals 


Angular  Rate  Residuals 


Fig.  9  Estimate  of  body-frame  angular  rates  using  perfect  OF  without  stable  flight 
constraints 


For  this  example  trajectory,  these  results  suggest  that  for  perfectly  known  OF,  the 
epipolar  constraint  36  in  combination  with  projectile  motion  constraints  37  appear 
sufficient  to  precisely  recover  v  and  w. 


Simulation  results  using  measured  OF  are  shown  in  Figs.  10  and  11.  Residuals  are 
shown  in  Fig.  12. 


State  Estimate  Performance  State  Estimate  Performance 


012345  012345 


time  [s]  time  [s] 

Fig.  10  Estimate  of  pitch  and  yaw  angles  of  attack  using  measured  OF 


Fig.  11  Estimate  of  body-frame  angular  rates  using  measured  OF 
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AoA  Residuals 


Fig.  12  Residuals  of  angles  of  attack  and  angular  rates  using  measured  OF 


The  increased  estimation  error  when  using  measured  OF  is  due  to  both  limited 
imaging  resolution,  as  well  as  mismatched  frame-to-frame  point  correspondences 
(tracking  error).  The  latter  is  easily  identified,  as  in  Fig.  13. 


Optical  Flow:  measured  vs.  true 

0.6  h .  . 


-0.61- .  . ^ 

-0.6  -0.4  -0.2  0.0  0.2  0.4  0.6 


■  true 

■  measured 


Fig.  13  Overlay  of  true  vs.  measured  flow  fields 

These  false  flow  vectors  clearly  violate  the  rigid-scene  assumption.  An  automated 
outlier  rejection  scheme  could  be  devised  by  demanding  smooth  flow  variation 
across  the  image  plane. 

7.  Conclusions  and  Future  Work 


Optical  flow-based  state  estimation  could  significantly  enhance  the  scope  and 
performance  of  future  guided  projectiles  by  mitigating  reliance  on  GPS  and 
augmenting  traditional  inertial  measurement  units  (IMUs)  and  other  navigational 
sensors.  Though  not  an  ideal  viewing  platform,  the  projectile  is  nonetheless 
uniquely  well-suited  as  an  observer  of  OF.  The  characteristically  elevated  vantage 
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point  and  expansive  view  justify  the  rigid-scene  assumption.  Stable  flight  dynamics 
and  forward  motion  also  serve  as  state  constraints. 

The  goal  of  this  work  has  been  to  adapt  the  mathematics  of  OF  geometry  to 
projectile  vision-based  navigation  and  present  the  smart  munitions  community  with 
an  integrated  modeling  and  simulation  framework  for  OF-based  state  estimation.  A 
flight-vision  model  has  been  presented  that  can  be  used  to  generate  example 
12-state  histories  with  specified  viewing  behavior,  along  with  corresponding 
synthetic  imagery,  exact  OF,  and  measured  OF. 

The  differential  epipolar  constraint  was  then  used  to  recover  translational  and 
rotational  velocity  vectors  v  and  oo  from  the  projected  flow  field,  and  performance 
was  evaluated  against  ground  truth.  State  estimation  was  further  improved  by 
imposing  stable-flight  constraints.  The  approach  was  validated  using  exact  OF  and 
then  applied  to  more  realistic  measured  flow  fields  derived  from  a  corresponding 
video  sequence. 

The  precision  of  OF  measurement,  though  not  a  primary  focus  of  this  study,  will 
likely  remain  the  primary  error  source.  More  sophisticated  measurement  techniques 
and/or  dedicated  OF-sensors  will  certainly  result  in  improved  estimation. 

Another  challenge  will  be  achieving  real-time  performance.  Speed  of  processing  is 
limited  by  imaging  resolution,  frame-rate,  and  computing  architecture,  as  well  as 
measurement  and  solution  algorithms.  Fundamentally,  the  required  (minimum) 
resolution  and  frame-rate  will  depend  on  the  projectile’s  rate  of  motion 
(primarily  (p )  and  range  to  scenery.  A  more  quantified  description  of  this  trade- 
space  will  help  to  parametrically  link  imaging  requirements  to  mission  profile. 

Finally,  scene  occlusion  is  particularly  problematic  for  high-arcing  trajectories 
and/or  narrow  FOV.  In  this  study,  the  imager  was  assumed  to  be  a  bore-sighted 
monocular  camera.  If  instead  the  camera  were  directed  at  some  off-axis  angle, 
projectile  spin  would  allow  for  periodic  state-updates  along  the  entire  trajectory. 
Likewise  an  array  of  2  or  more  imagers  could  allow  for  more  frequent  updates  and 
greater  scene  coverage. 
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Nomenclature 


{%%k} 

Body  frame  basis 

(x,y,z) 

Earth  frame  basis 

{x0,y0,z0} 

Initial  projectile  position,  Earth  frame  (m) 

{9V,  ipv] 

Pitch  and  yaw  angles  associated  projectile’s  translational  velocity 
vector (rad) 

[dp,  1pp[ 

Pitch  and  yaw  perturbation  angles  (rad) 

{Te*  r^j  rB,} 

Earth  frame,  inertial  body  frame,  rotating  body-frame 

{a,p} 

Pitch  and  yaw  angles  of  attack  (rad) 

{4>.W} 

Projectile  euler  angles:  roll,  pitch,  and  yaw;  Earth  frame  (rad) 

b  o 

Body  frame  origin  (projectile  position)  (m) 

eo 

Earth  frame  origin  (m) 

P 

Scene  point,  p  6  E3  (m) 

q 

Projected  scene  point,  q  e  E3  (m) 

r 

Projectile  position  vector  (m) 

V 

Projectile  translational  velocity  vector  (m/s) 

O) 

Projectile  angular  velocity  (rad/s) 

f 

Focal  length.  refer  to  the  effective  focal  lengths 

corresponding  to  the  (j,  k)  body  dimensions  (m) 

9 

Acceleration  of  gravity  (m/s2) 

t 

Flight  time  (s) 

At 

Time  laps  between  image  frames  (s) 

FOV 

Field  of  view  (rad) 

P 

Projection  matrix 

QE 

Launch  angle  above  horizon  (rad) 

R 

Rotation  matrix 

T 

Translation  matrix 
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